- What is reproducible research?
- Why do we care?
- Why reproducibility questions arise?
- The cost of reproducibility
- Reproducibility and statistics
- Current status of reproducibility
- What can we do?
Mikhail Dozmorov, Ph.D.
mikhail.dozmorov@vcuhealth.org
Reproducible research is the ultimate standard for strengthening scientific evidence by independent:
- Described drug response “gene signatures” in NCI60 cell lines
- Demonstrated these “signatures” correspond to patient-specific signatures and can be used to predict patient response to the drugs
- Described drug response “gene signatures” in NCI60 cell lines
- Demonstrated these “signatures” correspond to patient-specific signatures and can be used to predict patient response to the drugs
Published Replicated
... [3,] 1881_at 1882_g_at [4,] 31321_at 31322_at [5,] 31725_s_at 31726_at [6,] 32307_r_at 32308_r_at ...
http://www.nationalacademies.org/hmd/reports/2012/evolution-of-translational-omics.aspx
http://science.sciencemag.org/content/335/6076/1554.long
http://www.nature.com/nature/journal/v502/n7471/full/nature12564.html
“Reproducible research” “Retraction”
Human beings do not have very many natural defenses. We are not all that fast, and we are not all that strong. We do not have claws or fangs or body armor. We cannot spit venom. We cannot camouflage ourselves. And we cannot fly. Instead, we survive by means of our wits. Our minds are quick. We are wired to detect patterns and respond to opportunities and threats without much hesitation.
Our intuition about patterns in high dimensional data quickly drops with the increased dimensionality of the data
We rely on computation to uncover patterns
P values, the ‘gold standard’ of statistical validity, are not as reliable as many researchers assume.
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108
http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503
https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_policies.html
https://gds.nih.gov/03policy2.html
https://grants.nih.gov/grants/policy/model_organism/
https://www.nsf.gov/bfa/dias/policy/rcr.jsp
http://www.equator-network.org/ - over 300 reporting guidelines